IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers

High scalability and low operating cost make black-box protocol fuzzing a vital tool for discovering vulnerabilities in the firmware of IoT smart devices. However, it is still challenging to compare fuzzers due lack unified benchmark images, complete mutation seeds, comprehensive performance metrics, standardized evaluation framework. In this paper, we design implement IoTFuzzBench, scalable, modular, metric-driven automation framework evaluating devices comprehensively quantitatively. Specifically, IoTFuzzBench has so far included 14 real-world 30 verified vulnerabilities, seeds each vulnerability, 7 popular fuzzers, 5 categories complementary metrics. We deployed evaluated on all images vulnerabilities. The experimental results show that can not only provide fast, reliable, reproducible experiments, but also effectively evaluate ability fuzzer find differential different found total 13 out 30. None these outperform others This result demonstrates importance hope our findings ease burden scenarios, advancing more pragmatic benchmarking efforts.